Finding Public Opinion Manipulation Trolls in Bulgarian Online News Media

With the rise of social media, it became normal for people to read and follow other users' opinion. This created the opportunity for corporations, governments and others to distribute rumors, misinformation, and speculation and to use other dishonest practices to manipulate public opinion (Derczynski and Bontcheva , 2014). They could consistently use trolls (Cambria, Chandra and Sharma , 2010), write fake posts and comments in public forums, thus making veracity one of the challenges in digital social networking (Derczynski and Bontcheva , 2014). During the recent popular protest in Bulgaria in 2013 1 , social networks and news community forums became the main " battle grounds " between supporters and opponents of the government. In that period, there was notable censorship in the media, and many people who lived outside the capital did not really know what was actually happening. Moreover, there was a very notable presence of government supporters in Web forums. In series of leaked documents in the independent Bulgarian media Bivol, it became clear that the ruling party was using European parliament money to pay for hiring Internet trolls 2 3. In our work, we aim to find troll users and troll users' comments using several machine learning and natural language processing techniques. We first collect data about user profiles, comments and publications from the largest online media forum-that of Dnevnik.bg, where there is a notable presence of troll users. We build a Web crawler and several data extraction helpers for retrieving Web content and extracting structured data from the HTML content. We save the retrieved data in a relational database that reflects the natural data structure presented in the online media forum. We then use several database queries to retrieve statistical information to build user profiles and to retrieve users' comments as text and metadata. Using information and assumptions about troll vs. non-troll user behavior, we extract both statistics and text features and we build two classifiers using several configurations: one trained for finding troll vs. non-troll users and one for finding troll vs. non-troll comments. Our classifiers perform robust with accuracy of 90-96% for the user profiles' classifier and accuracy of 80-84% for the troll's comments classification. As a result of this work we have two independent papers accepted to CoNLL 2015 and RANLP 2015.